Addressing the Installation Bug in Building Sentencepiece by yeahjack · Pull Request #113 · zai-org/GLM-130B

yeahjack · 2023-04-02T06:12:19Z

Hi team,

I am sharing a log related to an issue with the package for sentencepiece. The package appears to fail to build when CMake is not installed first. Therefore, when setting up a new conda environment or installing packages from scratch, please make sure to install CMake before attempting to install sentencepiece.

I hope this log is helpful in resolving any issues related to the installation of sentencepiece.

Here is the log when I install it from a completely new conda environment:

  Building wheel for sentencepiece (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [80 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-311
      creating build/lib.linux-x86_64-cpython-311/sentencepiece
      copying src/sentencepiece/__init__.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
      copying src/sentencepiece/_version.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
      copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
      copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
      running build_ext
      /bin/sh: 1: pkg-config: not found
      Cloning into 'sentencepiece'...
      Note: switching to '58f256cf6f01bb86e6fa634a5cc560de5bd1667d'.
      
      You are in 'detached HEAD' state. You can look around, make experimental
      changes and commit them, and you can discard any commits you make in this
      state without impacting any branches by switching back to a branch.
      
      If you want to create a new branch to retain commits you create, you may
      do so (now or later) by using -c with the switch command. Example:
      
        git switch -c <new-branch-name>
      
      Or undo this operation with:
      
        git switch -
      
      Turn off this advice by setting config variable advice.detachedHead to false
      
      ./build_bundled.sh: 19: cmake: not found
      ./build_bundled.sh: 20: cmake: not found
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-ainsg3z2/sentencepiece_3bc691100aba437286ca44d76e09c437/setup.py", line 136, in <module>
          setup(
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/__init__.py", line 108, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/dist.py", line 1221, in run_command
          super().run_command(command)
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/wheel/bdist_wheel.py", line 343, in run
          self.run_command("build")
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/dist.py", line 1221, in run_command
          super().run_command(command)
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 131, in run
          self.run_command(cmd_name)
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/dist.py", line 1221, in run_command
          super().run_command(command)
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 84, in run
          _build_ext.run(self)
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
          self.build_extensions()
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
          self._build_extensions_serial()
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
          self.build_extension(ext)
        File "/tmp/pip-install-ainsg3z2/sentencepiece_3bc691100aba437286ca44d76e09c437/setup.py", line 89, in build_extension
          subprocess.check_call(['./build_bundled.sh', __version__])
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['./build_bundled.sh', '0.1.97']' returned non-zero exit status 127.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for sentencepiece
  Running setup.py clean for sentencepiece
  Building wheel for anykeystore (setup.py) ... done
  Created wheel for anykeystore: filename=anykeystore-0.2-py3-none-any.whl size=16814 sha256=7ccd544780f0b3ec4e0af12e17ac5d03a6249f681c679d6ee99ef616270f4482
  Stored in directory: /home/xuyijie/.cache/pip/wheels/1a/69/29/bb8eca19356ee2b9c96124c679c64ad107880b1e5160627efb
  Building wheel for pbkdf2 (setup.py) ... done
  Created wheel for pbkdf2: filename=pbkdf2-1.3-py3-none-any.whl size=5081 sha256=72d4ba81ce4cee11b7a8fbaecbba732e54c167252d627cf876a6b493af305ebe
  Stored in directory: /home/xuyijie/.cache/pip/wheels/69/9e/91/97e44b66d0a6a2ca5ed0fd4e1c2c082d504d926d46be0e9204
  Building wheel for lit (setup.py) ... done
  Created wheel for lit: filename=lit-16.0.0-py3-none-any.whl size=93586 sha256=faab572d58df366907496a09b089efecac2f5fdba2cf12b74ae2dd8c28d5bf85
  Stored in directory: /home/xuyijie/.cache/pip/wheels/32/7b/f9/056928d438450844917f9ee12eebcb6c77065c77ff3bf40dc6
Successfully built apex velruse cryptacular deepspeed anykeystore pbkdf2 lit
Failed to build sentencepiece
Installing collected packages: translationstring, tokenizers, sentencepiece, pytz, py-cpuinfo, pbkdf2, ninja, mpmath, lit, hjson, dataclass_wizard, cpm_kernels, cmake, anykeystore, zope.interface, zope.deprecation, xxhash, webob, venusian, urllib3, typing-extensions, tqdm, sympy, six, regex, pyyaml, psutil, protobuf, plaster, pillow, PasteDeploy, packaging, oauthlib, nvidia-nvtx-cu11, nvidia-nccl-cu11, nvidia-cusparse-cu11, nvidia-curand-cu11, nvidia-cufft-cu11, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cuda-cupti-cu11, nvidia-cublas-cu11, numpy, networkx, multidict, MarkupSafe, idna, hupper, greenlet, fsspec, frozenlist, filelock, dill, defusedxml, cryptacular, charset-normalizer, certifi, attrs, async-timeout, yarl, wtforms, transaction, tensorboardX, SQLAlchemy, scipy, requests, python3-openid, python-dateutil, pydantic, pyarrow, plaster-pastedeploy, nvidia-cusolver-cu11, nvidia-cudnn-cu11, multiprocess, jinja2, aiosignal, zope.sqlalchemy, wtforms-recaptcha, responses, requests-oauthlib, repoze.sendmail, pyramid, pandas, huggingface-hub, aiohttp, velruse, transformers, pyramid_mailer, datasets, apex, triton, torch, torchvision, deepspeed, SwissArmyTransformer, icetk
  Running setup.py install for sentencepiece ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for sentencepiece did not run successfully.
  │ exit code: 1
  ╰─> [69 lines of output]
      running install
      /home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-311
      creating build/lib.linux-x86_64-cpython-311/sentencepiece
      copying src/sentencepiece/__init__.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
      copying src/sentencepiece/_version.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
      copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
      copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
      running build_ext
      /bin/sh: 1: pkg-config: not found
      fatal: destination path 'sentencepiece' already exists and is not an empty directory.
      fatal: destination path 'sentencepiece' already exists and is not an empty directory.
      ./build_bundled.sh: 19: cmake: not found
      ./build_bundled.sh: 20: cmake: not found
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-ainsg3z2/sentencepiece_3bc691100aba437286ca44d76e09c437/setup.py", line 136, in <module>
          setup(
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/__init__.py", line 108, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/dist.py", line 1221, in run_command
          super().run_command(command)
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/command/install.py", line 68, in run
          return orig.install.run(self)
                 ^^^^^^^^^^^^^^^^^^^^^^
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/command/install.py", line 697, in run
          self.run_command('build')
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/dist.py", line 1221, in run_command
          super().run_command(command)
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 131, in run
          self.run_command(cmd_name)
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/dist.py", line 1221, in run_command
          super().run_command(command)
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 84, in run
          _build_ext.run(self)
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
          self.build_extensions()
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
          self._build_extensions_serial()
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
          self.build_extension(ext)
        File "/tmp/pip-install-ainsg3z2/sentencepiece_3bc691100aba437286ca44d76e09c437/setup.py", line 89, in build_extension
          subprocess.check_call(['./build_bundled.sh', __version__])
        File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['./build_bundled.sh', '0.1.97']' returned non-zero exit status 127.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> sentencepiece

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

After running conda install cmake and rerun pip install -r requirements.txt, it is resolved.

fix the bug of sentencepiece installation

3139a47

yeahjack changed the title ~~fix the bug of sentencepiece installation~~ Addressing the Installation Bug in Building Sentencepiece Apr 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Addressing the Installation Bug in Building Sentencepiece#113

Addressing the Installation Bug in Building Sentencepiece#113
yeahjack wants to merge 1 commit intozai-org:mainfrom
yeahjack:main

yeahjack commented Apr 2, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

yeahjack commented Apr 2, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant